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Method and Architecture for an 
Improved CMOS Color Image Sensor 

CLAIM OF PRIORITY 

5 This application claims priority based on the provisional application with 

serial No. 60/092,533 that was filed on July 13, 1998 with the U.S. Patent and Trademark 
Office entitled "Method and Apparatus for A CMOS Image Sensor With A Pixel 
Processor", which is hereby incorporated by reference for all purposes. 

FIELD OF THE INVENTION 

10 The present invention generally relates to image sensors and more 

particularly relates to a method and architecture for an improved CMOS color image 
sensor having a pixel processor integrated therewith. The improved CMOS color image 
sensor provides dynamic access to photo-sensors therein for improved signal throughput 
and, along with the pixel processor, for tasks including preview, digital zoom, shuttle 

15 control, color balance, and gray-scale readout. 

BACKGROUND OF THE INVENTION 

Digital photography is one of the most exciting technologies that emerged 
in the past ten years. With the appropriate hardware and software (and a little 
knowledge), anyone can put the principles of digital photography to work. Digital still 
20 and video cameras are on the cutting edge of digital photography. Recent product 

introductions, technological advancements, and price cuts, along with the emergence of 
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email and the World Wide Web, have helped make the digital cameras the hottest new 
category of consumer electronics products. 

Unlike traditional cameras that use film to store an image, digital cameras 
use a photosensitive device called an image sensor. Until recently, the dominant and 
nearly only solid-state image sensor technology was the charge-coupled device (CCD). 
However, most microprocessors, logic circuits, ASICs (application-specific integrated 
circuits), and memory circuits are based on Complementary Metal Oxide Semiconductor 
(CMOS) technology. Logically, CMOS will be used for image sensors in a rapidly 
expanding sphere of applications. 

Several important factors have contributed to the emergence of CMOS 
image sensor arrays at this time rather than 10 years ago. The primary factor is a recent 
demand by customers for portable, low power, miniaturized digital imaging systems. A 
second important factor is that current CMOS technology offers submicron feature sizes 
and low defect and contamination levels, respectively permitting cost-effective pixel sizes 
and low junction leakage (or dark) current. In addition, threshold voltage control and 
uniformity is stable and reproducible. The third important factor is that new circuit 
techniques have been invented or have been adopted from CCD signal processing that 
permit both low noise and high-dynamic-range imaging that is competitive with the best 
CCDs. 

A CMOS image sensor array comprises millions of photocells or photo 
sensors. Each of the photo sensors produces an electronic signal representing the 
intensity of the light that reflects from a target and falls on it by accumulating a charge; 
generally the more light, the higher the charge is. After the light integration, the 
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electronic signals from all of the photo sensors are readout and then digitized through an 
analog-to-digital converter to produce digital image of the target. 

Figure 1 illustrates an exemplary use of a CMOS image sensor in an 
imaging sensing system 100 such as a digital camera. The image sensing system 100 
comprises an optical system 132 for focusing an image onto image sensor array device 
130. Readout circuitry 133 reads an electronic image information from the image sensor 
array 130. The readout circuitry 133 passes analog image signal representing the target 
image is to an analog-to-digital converter (A/D) 140 for digitization. 

The A/D converter 140 digitizes the analog image signal from the image 
sensor 130 to create a digitized signal that can be appropriately and subsequently stored in 
memory 150. Typically, an imaging system 100 further comprises a digital signal 
processing circuitry 160 that performs various processing tasks to put the digital image 
into a suitable form. For example, the digital signal processing circuitry 160 may adjust, 
correct, preprocess and compress the digitized signal to eventually output an appropriate 
digital image or signal. In the embodiment of Figure 1, the digital signal processing 
circuitry 160 performs operations necessary to place the image signal into suitable form 
for display onto display 171 using display driver 170. 

As shown in Figure 1, traditional image sensor arrays could only output 
image signals that required further signal conditioning before such image signals could be 
used. For example, the imaging system 100 required digital signal processing circuitry 
160 to processes the image signal before it could be passed to the display system. To 
simplify the design of an image system, it would be desirable to eliminate or reduce the 
need for a digital signal processor in an imaging system. 
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SUMMARY OF THE INVENTION 

According to the principles of this invention, an improved CMOS image 
sensor array is disclosed. The improved CMOS image sensor array comprises a pair of 
controllable colunrn and row decoders, a signal conditioning circuit and a pixel processor 
5 in addition to an array of photo sensors. With the pair of controllable column and row 
decoders, the improved CMOS image sensor array can selectively and dynamically access 
particular desired photo sensors. Thus, applications that do not require the full set of 
signals from the entire array of photo sensors can access the desired photo sensors. Along 
with an on-chip memory, digitized signals from the selected photo sensors can be 
Q 10 processed in the pixel processor for auto focus, pixel decimation, pixel interpolation, 
iy pixel averaging, data conversation, and compression. The conditioning circuit comprises 

fy a correlated double sampling circuit and a programmable gain amplifier. The 

i. h combination of the two circuits makes it possible to provide color balancing and auto- 

exposure control in a more precise manner and further facilitates a wider dynamic range 
tf. 15 by the analog-to-digital converters. 



The advantages of the present invention are numerous. Different 
embodiments or implementations including methods and circuit architecture may yield 
one or more of the following advantages. By selectively and dynamically accessing 
signals in the photo sensors, transmission of only a smaller set of useful signals reduces 
20 the bandwidth of the buses that otherwise must be high enough to accommodate all the 
signals readout from the photo sensors. Another important feature of the present 
invention is the high signal throughput for applications that require only a small selected 
set of pixel signals from the photo sensor array. For example, a small number of pixel 
signals may be very quickly read out and processed for applications such as image 
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preview, color balancing, exposure setting, and auto focus. Another important feature in 
the present invention is the use of the pixel processor that can be configured to perform 
many internal processing of the image data. Consequently, the design complexity of an 
overall imaging system using the present invention is considerably reduced and the 
5 performance thereof is substantially increased. For example, the image sensor array of 
the present invention can output digital image information that can be directly used to 
drive a digital image display. 

Other objects and advantage together with the foregoing are attained in the 
exercise of the invention in the following description and resulting in the embodiment 
C3 10 illustrated in the accompanying drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



These and other features, aspects, and advantages of the present invention will 
become better understood with regard to the following description, appended claims, and 
accompanying drawings where: 

Figure 1 shows an exemplary use of a CMOS image sensor in an imaging 
sensing system that comprises an image sensor array device and an optical system; 

Figure 2A shows a schematic diagram of a system that uses involving a 
digital imaging system; 

Figure 2B illustrates a Bayer pattern two-dimensional image sensor array 
that comprises a plurality of photo sensors fabricated on a Complementary Metal-Oxide 
Semiconductor (CMOS) device; 

Figure 2C illustrates a typical CMOS image sensor array; 

Figure 3A shows a functional block diagram of an improved image sensor 
device according to one embodiment of the present invention; 

Figure 3B depicts the Bayer arrangement and each column of the photo 
sensors is coupled to a readout bus; 

Figure 4 shows the reduced design complexity of an imaging system 
employing the improved image sensor of the present invention; 

Figure 5A shows an image sensor having 1024 by 768 photo sensors, 
wherein each of the photo sensors is addressed by a pair of column address decoder and 
row address decoder; 
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Figure 5B illustrates a highlighted serial pixel decimation signal for the 
image sensor of Figure 5A; 

Figure 6A illustrates a flowchart to illustrate the process for reading out 
electronic signals from a region of interest (ROI) for auto focus, digital zoom and other 
similar applications; 

Figure 6B illustrates an exemplary approach based on derivative 
calculations for the pixel processor to cause the imaging system to focus on the target 
consequently to produce in-focus images; 

Figure 7A illustrates that an image sensor is focused on a house but the 
user is interested in viewing the front window portion; 

Figure 7B shows the decimation process flowchart according to one 
embodiment of the present invention; 

Figure 7C shows an example of a small portion of signals from the image 
sensor being enlarged by the interpolation process; 

Figure 8A illustrates a first example signals from an array of photo 
sensors are being decimated; 

Figure 8B illustrates a conceptual diagram of pixels being decimated; 

Figure 9A illustrates four neighboring pixel signals from the R, G, and B 
photo sensors being co-sited. 

Figure 9B illustrates four neighboring pixel signals from the R, G, and B 
photo sensors being co-sited. 
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Figure 10 illustrates pixel signals from the R, G, and B photo sensors in 
array of photo sensors are being converted to gray-scale intensities. 



m 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 



In the following detailed description of the present invention, numerous 
specific details are set forth in order to provide a thorough understanding of the present 
invention. However, it will become obvious to those skilled in the art that the present 
invention may be practiced without these specific details. The description and 
representation herein are the common means used by those experienced or skilled in the 
art to most effectively convey the substance of their work to others skilled in the art. In 
other instances, well-known methods, procedures, components, and circuitry have not 
been described in detail to obscuring aspects of the present invention. 

Color Images and CMOS Sensors 

Referring now to the drawings, in which like numerals refer to like parts 
throughout the several views. Figure 2A shows a schematic diagram of a system 
involving a digital imaging system 200. Under either natural or man-made illumination, 
optics unit 202 collects light reflected from an object 104 and focuses the reflected light 
onto an image sensor 204. The imaging system 200 subsequently produces digital image 
data 108 that may be used to display a representation of object 104 on a computer 206. 
Alternatively, imaging system 200 may produce video signals (sequence of temporal 
images) 210 that may be displayed on a television system 208. 

Depending on the applications, the imaging system 200 (that may be a 
digital still camera or a digital video camera) may produce a gray-scale or color image. In 
the case of a gray-scale image, digital image data 108 comprises a plurality of pixels 
wherein each pixel is represented by a numerical value. The pixel numerical value 
represents the intensity of incident light impinged upon the image sensor 204 in imaging 
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system 200 from a corresponding location on object 104. For example, if the object 104 
is an 8.5-inch by 1 1-inch paper; resultant image may have a size 850 by 1 100 pixels and 
is represented in 8-bit format wherein each digital pixel value ranges from 0 to 255. This 
means that each square inch of scanning object 210 is represented by 100 by 100 pixels. 
5 If all the pixels from a particular square inch report a value of 255, then the square inch in 
object 210 is white. Similarly, if all the pixels for a square inch report an output value of 
zero ("0"), then square inch in object 104 is dark. It can be understood that any pixels 
having a value between 0 and 255, i.e. the gray scale, represent the light intensity 
variations of the object 104. 

10 When the imaging system 200 reproduces colors of the object 104, the 

digital image data 108 from imaging system 200 typically comprises a set of three 
different light intensity images for three different colors: red, green, and blue. Each of the 
three light intensity images being an array or matrix of pixels having a value between 0 to 
255 when presented in an 8-bit precision format. When the three pixel matrixes are 

15 combined, each color pixel C(x, y) in color image 108 is a vector pixel that may be 
expressed as follows: 



where (x, y) are coordinates of an image pixel and C refers to color image 108 and R, G 
and B are the respective light intensity values for red, green, and blue light. If a cluster S 
of corresponding pixels in each of the three intensity images have an identical value, 
20 namely R(x, y) = G(x, y) = B(x, y), then spot S on object 104 is colorless. Spot S would 
appear as a gray that is visually between dark (0) to white (255). Conversely if a cluster 
of pixels S' has different color intensity value, i.e. R(x, y) -t- G(x, y) B(x, y), the spot S' 
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on object 104 would be visually colorful. For example, a pure red, green, or blue pixels 
are expressed as C (x, y) = [255 0 0]'^, C (x, y) = [0 255 0]'^, or C (x, y) = [0 0 255]^^, 
respectively. To ensure that object 104 can be exactly reproduced as a color image, the 
imaging system 200 must be well designed to accurately capture the three different color 
5 intensity arrays. The three different color intensity arrays are then combined to reproduce 
the colors of object 104 in the resultant color image. 

A CMOS Image Sensor Array 

Figure 2B illustrates a block diagram of a typical prior art two- 
dimensional Complementary Metal-Oxide Semiconductor (CMOS) image sensor array. 
10 Each element in the array (such as array elements 221, 222, and 223) is an individual 

CMOS image sensor circuit. The individual CMOS image sensor circuits are also known 
as pixel circuits. In a gray scale image sensor, each photo sensor captures light without 
regard to light color. 

In a color image sensor array, as illustrated in Figure 2B, each of the 
15 individual CMOS image sensor circuits may be covered by a selectively transmissive 
filter. According to one embodiment, there are three selectively transmissive filters, red 
(R), green (G), and blue (B). Thus, the individual CMOS image sensor circuits 
superimposed respectively with a red, a green, or a green filter, are reactive to the red, 
green or blue light components of the incoming light. Therefore, the image sensor array 
20 204 of Figure 2B produces R, G, and B intensity signals. The photo sensors are highly 
sensitive to light and each produces a proportional electronic signal with respect to the 
strength of the incident light. 

Figure 2C illustrates a more detailed embodiment of a CMOS image 
sensor array. The CMOS image sensor array of Figure 2C is controlled by row decoder 
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281 and a column decoder 282. The row decoder 281 and column decoder 282 select a 
particular CMOS image sensor circuit to be accessed. 

The output of an activated CMOS image sensor circuit is carried down a 
column output line to a sense and hold circuit 283. The sense and hold circuit 283 senses 
the voltage value of an activated CMOS image sensor circuit. The sensed voltage value 
is converted into a digital value by an analog to digital converter 284. The digital value 
can be read out of analog to digital converter 284 by an outside unit. 

Improved CMOS Image Sensor Device 

Figure 3 A shows a functional block diagram of an improved image sensor 
device 300 according to one embodiment of the present invention. As can be seen in 
Figure 3A, th^ improved image sensor device 300 integrates several functions of a digital 
imaging system onto a single integrated circuit die. 

When the image sensor 300 is used in the imaging system 100 of Figure 1, 
the design complexity of the imaging system 100 can be significantly simplified as 
illustrated in Figure 4. Specifically, when an imaging system is implemented with the 
improved image sensor device 300, the imaging system can use the output signals from 
the improved image sensor device 300 directly. Thus, the improved image sensor device 
300 may be coupled directly to a display driver 402 for display on a display screen 472. 
A digital signal processor (DSP) 404 may be coupled to the improved image sensor 
device 300 for performing more sophisticated processing tasks. For example, the digital 
signal processor (DSP) 404 may perform object extraction and recognition in robotic 
vision applications, video streaming for Internet video applications, image compression, 
image manipulation, and other image processing operations. 
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The Photo Sensor Array 

The image sensor device 300 comprises an array of photo sensors 302 
superimposed with red, green and blue filters. In one embodiment, the arrangement of R, 
G, and B photo sensors is conformed to the Bayer arrangement. Furthermore, each 
5 colunm of photo sensors is coupled to a readout column bus 350-1, 350-2, ... 350-n as 
shown in Figure 3B. As such, electronic image signals from an entire row of photo 
sensors can be readout in parallel. According to one embodiment, the image sensor 
device 300 has an array of 1024 by 768 photo sensors and hence there are 1024 readout 
buses. Under clock signals from a timing generator 316, it takes 768 cycles to serially 
10 read out all of the electronic signals in the image sensor 300 after the light integration 

11 Sir 

^0 process. 



fy Photo Sensor Sampling 

As a part of an internal auto-calibration process, the outputs of columns of 
the photo sensors 302 are carried down the readout buses 350, respectively, to a bank of 
15 correlated double sampling circuits (CDS) 308 that sample the signals. In a preferred 
embodiment, the sampling method of the correlated double sampling circuits (CDS) 308 
removes fixed pattern noise from the electronic signals. The fixed pattern noise is unique 
to CMOS image sensors and considered one of the main disadvantages in comparison 
with CCD image censors. In a prior art method, canceling the fixed pattem noise can be 
20 performed by off-chip storage of the offset values, obtained by reading the output of the 
photo sensors while they are reset. That method, however, requires a large amount of 
memory for storing the entire array of offset information. 

The image sensor device 300 uses a correlated double sampling circuit 308 
that is an efficient on-chip method. During a readout process, two samples are taken: a 
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first sample when a photo sensor is in a reset state and a second sample when the 
electronic signal from the photo sensor has been transferred to the read-out node. 

Analog Pixel Processor 

After sampling the image sensor device with a correlated double sampling 
5 circuit 308, the analog signals can be processed by an analog pixel processor (APP) bank 
310. The analog pixel processor can perform many different types of pixel processing in 
the analog domain. For example, the analog pixel processor (APP) bank 310 may 
perform edge detection, motion processing, or color balancing. 

In one embodiment, one analog pixel processor is shared for every N 
10 columns of the photo sensor array. For example, there may be one analog pixel processor 



\jl for every eight columns from the photo sensor array. In such an embodiment, the analog 

fU 

pixel processor is shared by the eight columns in a time division multiplexing manner. 
This allows a sophisticated analog pixel processor to be created even when the pixel pitch 
becomes very narrow. 

15 In one embodiment, the analog pixel processor comprises a programmable 

gain amplifier for color balancing. Energy from red, green, and blue light is not absorbed 
equally by the photo sensors in the photo sensor array 302. Thus, the integration values 
from red, green, and blue photo sensors must be equalized before being used within an 
image. Prior art imaging systems performed such color equalization in the digital domain 
20 with a digital signal processor long after the photo integration values had been read from 
the image sensor array. However, such systems lost valuable information from the 
weaker signals. Specifically, the dynamic range of weaker signals is lost since the 
digitization range must be set to capture the signal with the largest dynamic range. 
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One embodiment of an image sensor device built according to the 
teachings of the present invention uses the analog pixel processor to perform analog color 
balancing to the integration signals from the image sensor before the integration signals 
are digitized. Specifically, a programmable gain amplifier (PGA) bank amplifies the 
signals sampled by the correlated double sampling circuits (CDS) 308. The 
programmable gain amplifiers can amplify the incoming analog signals with a gain that 
depends on the measurement from the correlated double sampling circuit 308. In such a 
manner, the analog electronic signals are properly adjusted before being digitized in one 
of the analog-to-digital converters (ADC) 312. 

In other words, after an analog integration electronic signal from a photo 
sensor is read out, the analog integration electronic signal is first measured and then 
adjusted accordingly before it is sent to an ADC for digitization. This technique results in 
a maximum dynamdc range by allowing complete use of a full input range of the ADC 
bank 312. No dynamic range is lost due to offset in any of the three blocks 308, 310 and 
312. By having a wide programmable gain amplifier gain available, the imaging system 
can operate in a larger range of available light, allowing the analog-to-digital converter 
(ADC) bank 312 to utilize all of its resolution for each sample. 

In the gain amplifiers of many prior art image sensor arrays, all different 
pixels are amplified using independent controllable amplifiers for all three pixel colors. 
In such embodiments, green pixels are amplified with a gain of 1 .0 (unity gain), red pixels 
are amplified with a gain of 0.8 to 1.2, and blue pixels are amplified with a gain of 2.5 to 
3.0. 

In one embodiment of the programmable gain amplifier (PGA) bank 310, 
the programmable gain amplifier amplifies the green pixels with a predetermined fixed 
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gain value. In one embodiment, the green pixels are amplified with a gain of 1.2. The 
red and blue pixels are then amplified based upon the amplification gain of the green 
pixel. In one embodiment, red pixels are amplified with a gain of 1.0 to 1.5 and blue 
pixels are amplified with a gain of 3.0 to 3.5. Thus, the red and blue amplification values 
are variable and are normalized to the green value. 

It should be noted that each row of pixels from a Bayer pattem photo 
sensor array will consists of Red and Green pixels or Blue and Green pixels. Thus, the 
amplification values of every other programmable amplifier in the analog pixel processor 
310 may be the same. 

Analog to Digital Conversion 

After the analog pixel processor bank 310 has amplified the analog 
integration values, an analog-to-digital converter (ADC) bank 312 digitizes the signals. 
The output of the analog-to-digital converter (ADC) bank 312 is a set of n-bit digital 
values for each sampled pixel. 

As with the analog pixel processor bank 310, the analog-to-digital 
converter (ADC) bank 312 may be multiplexed. Specifically, there may be only one 
analog-to-digital converter device for each N columns in the photo sensor array. Ideally, 
the number of analog-to-digital converter devices should equal the number of analog 
pixel processors. In this manner, the two units may be organized in a pipeline fashion. 

By multiplexing the analog pixel processor bank 310 and the analog-to- 
digital converter (ADC) bank 312, the architecture becomes very scalable. The 
architecture is scalable in two different manners. First, the size of the photo-sensor array 
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can be increased linearly by adding one analog pixel processor and the analog-to-digital 
converter device for every N columns. 

Second, the architecture can be scaled according to finer pitches. For 
example, if the pixel pitch width decreases from 8 microns to 4 microns, then the 

5 multiplexing just needs to be adjusted if the same sized analog pixel processor and 

analog-to-digital converter device are used. For example, an 8 to 1 multiplexor would be 
replaced by a 16 to 1 multiplexor. In such a system, the frame rate would decrease unless 
the clock speed of the analog pixel processor and the analog-to-digital converter device 
were increased. Furthermore, additional current would need to be supplied to the faster 

10 operating analog pixel processor and the analog-to-digital converter device. 



yi Pixel Processor 



Resultant digital signals (pixels) from the analog-to-digital converter 
(ADC) bank 312 are coupled to a pixel processor 314. The pixel processor 314 may 
perform various tasks on the pixels as to be described in detail below. Depending on 
15 applications, the output of the pixel processor 314 may be static image data for still 



W digital camera applications or a stream of image data for video display. 



In a simple embodiment, the pixel processor is a register controlled state 
machine that comprises set of line buffers, associated arithmetic and logic unit (ALU) 
circuitry, and the control circuitry. The arithmetic and logic unit (ALU) circuitry is 
20 controlled by a set of control registers 320 that determine how the pixel processor 314 
operates. Using the digital pixel information in the line buffers and the arithmetic and 
logic unit (ALU) circuitry, the pixel processor 314 may perform many different arithmetic 
and logical operations on the pixels. By using several line buffers as a type of memory. 
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many filtering operations can be performed by logically combining temporally different 
versions of the same data. 

In a more sophisticated version, the pixel may comprise a processor unit 
that accesses programs in on-chip memory 317. On-chip memory 317 is primarily loaded 
with programs that are executed along with the pixel processor 314 to achieve desired 
results. For example, the output of a sophisticate pixel processor 314 may be in a 
compressed format such as JPEG (Joint Photographic Experts Group). Altematively, the 
pixel processor 314 may perform full or partial compression using MPEG, GIF, or 
wavelet compression systems. 

In the simpler embodiment, the pixel processor 314 may perform just a 
few of the initial steps of a compression system. For example, the line buffer system may 
process eight adjacent pixels for performing the early steps of JPEG compression. An 
image processor (a microprocessor or Digital Signal Processor) is thus given a "head 
start" on the JPEG processing. The detailed description of the processing by the pixel 
processor 314 in conjunction with the programs in memory 317 will be provided below. 

The pixel processor is programmable such that the processing performed 
by the pixel processor 314 will vary depending on the particular application. In one 
embodiment, the pixel processor 314 may be controlled using a set of control registers 
320. The control registers may store codes defining the processing to be performed and 
processing parameters. 

The pixel processor 314 is comprised, in part, of a number of pixel data 
rows 321. The pixel data rows 321 contain pixel data that has been read out from the 
photo sensor array 301. The pixel processor 314 can combined the various pixel data 
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rows 321 arithmetically and logically. The number of pixel data rows 321 within a 
particular pixel processor 314 embodiment will be based upon the pixel processing 
operations that need to be performed. 

The improved CMOS image sensor device 300 includes a timing generator 
316 that generates the clock signals and other timing references needed by the remainder 
of the improved CMOS image sensor device 500. To orchestrate the operation of all the 
different subsystems in the improved CMOS image sensor circuit 300, the improved 
CMOS image sensor circuit 300 includes control logic 218. 

The image data from the pixel processor 314 may be directly streamed out 
of the pixel processor 314 into another device. In a different embodiment, the output 
digital signals from the pixel processor 314 are passed to an input/out (I/O) interface 318. 
The input/out (I/O) interface 318 facilitates the transmission of the digital signals out of 
the image sensor device 300 to an external part that may be a digital signal processor or a 
display driver. In either case, the control registers 320 can be progranamed to determine 
how the pixel processor 314 or I/O interface 318 output the data. Specifically, the pixel 
processor 314 or I/O interface 318 may perform sub-sampling by outputting only a 
portion of the available data. According to one embodiment, the pixel processor 314, 
memory 317 and I/O interface 318 are monolithically integrated with the array of photo 
sensors 302. 

Window Gating 

Image processing is often done on small areas of a larger overall image 
array. For example, an auto-focus routine is often performed only used the central "focus 
area" of an image. Furthermore, a small area of an image containing white pixels may be 
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used to perform "white balancing" in a video camera application. To help indicate the 
data for such routines is being output, the present invention introduces "window gating". 
Window gating is used to identify a defined subset of pixels from a larger pixel area. 
Two different window-gating systems are introduced: random access window gating and 
serial highlighted window gating 

Random Access Window Gating 

Unlike Charge Coupled Device (CCD) image sensors in which electronic 
signals generated after the light integration must be all sequentially readout, the dynamic 
access provided by the CMOS image sensor device 300 of the present invention permits 
selective readout. In other words, signals in a particular area of photo sensors can be read 
out without reading out all the signals in the sensor. This feature is important to a 
number of applications that demand high signal throughput when the spatial resolution of 
the image sensor is increased. For example, it would take a considerable time to readout 
all 786,432 pixel signals from an image sensor having 1024 by 768 photo sensors thereon 
while not all of the signals are necessary for determining exposure time and digital zoom. 

Referring again to Figure 3A, there is a pair of colunm address decoder 
304 and row address decoder 306 in the image sensor device 300. With the column 
address decoder 304 and the row address decoder 306, each of the photo sensors 302 can 
be selectively accessed. One of the features in the present invention is to utilize the 
column address decoder 304 and row address decoder 306 to provide an effective readout 
from the image sensor for auto focus, color balance and exposure control. 

Referring to Figure 5A, there is an image sensor 500 having 1024 by 768 
photo sensors thereon and each of the photo sensors is addressed by a pair of colunm 
address decoder and row address decoder. A subarea or region of interest (ROI) 502 is 
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defined by parameters that are dependent on the applications. For example, for an auto 
focus application the region of interest (ROI) 502 is typically small (100 by 100 or 200 by 
200) and centered in the image sensor array 500. Therefore, the column address decoder 
produces column address signals identifying the photo sensors located on columns from 
5 462 to 561 and the row address decoder produces row address signals identifying the 
photo sensors located on rows from 334 to 433. The colunm address signals 504 and the 
row address signals 506 illustrate that region of interest (ROI) 502 is being defined and 
only the electronic signals in ROI 502 will be readout. It should be noted that the entire 
array of image sensor array 500 functions in the light integration process. Hence, every 
10 photo sensor in the image sensor array 500 has collected charges. However, only the 

electronic signals in the photo sensors within the defined window gating signals, namely 
the column address signals 504 and the row address signals 506, are readout. 
Consequently, the readout process after the light integration process is much faster. 



Highlighted Serial Window Gating 
15 Another method of identifying a desired pixel subarea is to use highlighted 

serial window gating. Highlighted serial window gating operates by signaling when 
Q pixels from the desired subarea are being output when an entire pixel array is output. For 

example, referring to Figure 5A a pixel array 500 is illustrated. Within the pixel array 
500, a smaller desired subarea 502 to be processed is identified with by a rectangle. In 
20 one embodiment of the present invention, the subarea 1010 is identified by a first row, a 
number of rows, a first colunm, and a number of columns. These parameters can be 
entered into control registers 320 that control the pixel processor 314 of the present 
invention. 

To inform another circuit coupled to the photo sensor device 300 as to 
25 when the pixels in the identified subarea 502 are being output, the present invention uses 
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a window gate signal. The window gate signal is normally inactive. Thus, when the first 
set of rows of pixel array 500 are scanned, the window gate signal remains inactive as 
illustrated by the window gate signals 551. However, when the improved photo sensor 
device 300 of the present invention is scanning out pixels that are within the identified 
subarea 502, the window gate signal is driven active as illustrated by window gate signals 
553. After the photo sensor device 300 has scanned past the pixels of the identified 
subarea 502, the window gate signal is inactive as illustrated by window gate signals 555. 

Using the window gate signal, an area in the center of the image array can 
be set as the subarea to be identified in a digital camera application. Then, when the 
autofocus routine needs to be performed, the pixels in the focus area will be identified to 
a digital signal processor using the windows gate signal. 

It is noted that the window gating signals do not have to define a region of 
interest (ROI) in the center of the image sensor. Furthermore, there can be one or more 
region of interest (ROI) in the image sensor depending on a specific application. 

Figure 6A illustrates a flowchart to illustrate the process for reading out 
electronic signals from a region of interest (ROI) for auto focus, digital zoom and other 
similar applications. At block 602, before the light integration process, the image sensor 
device needs a number of parameters to set it up for proper operations. Examples of the 
parameters include initial electronic shutter speed, initial exposure time, and initial white 
balance. It is assumed the imaging system is powered on and triggered to start the light 
integration process. With the parameters, particularly the boundary parameters for one or 
more region of interest (ROI), loaded at block 604, the image sensor integrates incident 
light for a defined time to produce an image of a target at block 606. Since the captured 
image data is for the determining if the imaging system has been properly adjusted, only a 
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small portion image data in the center area of the image sensor is needed. At block 608, 
the window gating signals are determined with respect to the boundary parameters. In a 
random access system, the window gating signals cause the column and row address 
signals to address only the desired portion of the array. In a highlighted serial window 

5 gating system, the window-gating signal is active when the desired region is scanned. At 
block 610, signals only in a small portion of the center area of the image sensor addressed 
by the column and row address signals are readout to column buses. It should be noted 
that the order of blocks 606 and 608 are interchangeable. As illustrated in Figure 6A, if 
there are no boundary parameters loaded at block 602, all signals in the image sensor will 

10 be readout at block 610. 

IfCS? 

tisr 

|y At block 612, the readout signals are processed by the CDS bank and the 

in 

fu PGA bank to produce a set of optimal image signals. At step 614, the ADC unit digitizes 

the analog signals to produce a set of digital signals that represent the area or areas 
'J defined by the boundary parameters. The set of digital signals is then processed in the 

15 pixel processor at block 616. Depending on the actual use of the set of digital signals, the 
pixel processor is configured to produce appropriate outputs. 



t 



According to one embodiment, one output from the pixel processor is for 
an auto focus feature. The pixel processor executes a set of instructions to determine if 
the incoming set of digital signals possesses sharp edges. Figure 6B illustrates an 
20 exemplary approach based on derivative calculations for the pixel processor to cause the 
imaging system to focus on the target consequently to produce in-focus images. It is 
assumed that the readout signals represent an image 630 of two contrast areas, thereby 
there is an edge between the two contrast areas. If the imaging system is out of focus, a 
single row signal 632 across the two regions tends to be monotonic and the pixel 
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processor can detect from the corresponding derivative signal 633 that the imaging 
system was out of focus and the output is forwarded to block 604 of Figure 6A to repeat 
blocks 606, 608, 610, 612 and 616 with a new setting for the focus. 

Another set of digital signals are then generated under the new setting for 
the focus, one of the signals is represented by a row signal 634. This time the row signal 
634 possesses certain discontinuity caused by the edge between the two contrast areas. 
The output of the pixel processor based on the corresponding derivative signal 635 causes 
the imaging system to further adjust the focus thereof and is forwarded to block 604 to 
start blocks 606, 608, 610, 612 and 616 with another new setting for the focus. 

Another set of digital signals is generated, one of which is represented by a 
row signal 636. The pixel processor can detect from the corresponding derivative signal 
that the imaging system is now in focus as the corresponding derivative signal possesses a 
sharp pulse resulting from a sharp edge. It is clear from the context that the focusing 
processing can be repeated until a determination passes a threshold. The output of the 
pixel processor may be set to cause a generation of an audible sound so that a user of the 
system is notified of the status and proceeds to capture images of the target. When it is 
ensured that the imaging system is in focus of the target, the image sensor is ready to 
produce a full image of the target. 

In a digital camera, the pre-focus process may be triggered by a half- 
activation of the shuttle button. In a video camera, the pre-focus process may be 
automatically started when a user presses a "On" or "record" button and before the sensor 
produces sequence of images. In either application, the auto focus process must be fast 
enough that the user experiences virtually no latency in capturing a desired target. Those 
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skilled in the art will appreciate that a fast speed is achieved by selectively reading out a 
small subarea of the full image sensor array and processing just that small area. 

It should be pointed that the example in Figure 6B is for illustration only. 
Many other approaches may be used by the pixel processor for auto focus application. 
The important feature of the present invention is the high signal throughput, hence 
considerable high speed for those applications that require a small set of image data. 
Reading a full set of image data from the image sensor requires a lengthy time that may 
cause jitter feeling to the user. 

Pixel Interpolation 

Digital zoom is an application that uses a small set of signals from an 
image sensor array but displays that small set of signals as a full image. Figure 7A 
illustrates an image of a house captured by an image sensor 702. However, the user is 
interested in viewing the front window portion. A mechanism is provided for the user to 
move a window 704 over the region of interest (ROI). The size of the window 704 is 
adjustable by the user to enclose the entire region (a front window of the house) as shown 
in Figure 7A. After an activation of a trigger, only signals corresponding to the ROI are 
readout and then processed in the pixel processor. The ROI is then properly displayed on 
a screen 706 in which only the front window of the house is displayed as illustrated in 
Figure 7A. 

Figure 7B shows the process flowchart according to one embodiment of 
the present invention. At step 710, the image sensor is ready for the light integration 
process but waits for a triggering signal that may come from a shutter release of a digital 
camera or "On" button from a video camera. Upon being triggered, the image sensor 
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starts the light integration process at block 712, accumulating photons of incident light for 
a predefined exposure time. To view the captured image, the signals in the image sensor 
are read out in an appropriate manner for a display for preview at step 714. A preview 
display is typically small and low resolution. For example, a typical pre-view screen may 
be two or three inches along its diagonal with a resolution of 70 dots-per-inch (DPI). A 
decimation process, as explained below, selectively reads out a limited number of pixels 
for display on the preview display. 

At step 716, the user defines a window by moving and adjusting the 
window around a region of interest (ROI), which inherently defines the boundary 
parameters and thus the window gating signals. As soon as the user is done with the 
region of interest, the signals in the photo sensor array corresponding to the region of 
interest are readout at step 718 with respect to the window gating signals. At step 720, 
the readout signals are processed by the CDS bank and the PGA bank. At step 722, the 
ADC bank digitizes the analog signals to produce a set of digital signals representing the 
region of interest defined by the boundary parameters. 

The set of digital signals is then processed in the pixel processor at step 
724. An interpolation methodology is implemented in the pixel processor to expand the 
limited pixel data. According to one embodiment, a neighbor linear interpolation 
approach is used. Specifically, a "missing" pixel is created by interpolating from its 
neighbor pixels. Those skilled in the art understand that there are other interpolating 
methods that may be implemented in the pixel processor. 

Figure 7C conceptually illustrates an example of a small portion of 
signals 754 from the image sensor array being enlarged by the interpolation process. 
After the image 750 is displayed, the user defines the region of interest by moving and 
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adjusting the window 752 to surround it. The addresses identifying the corresponding 
photo sensors 754 on the image sensor array are inherently determined and the signals 
thereof are correspondingly read out. The digital version of the signals is subsequently 
processed in the pixel processor for interpolation. In this example, an interpolated image 
5 256 has spatial resolution 4 times as large as the original corresponding photo sensors 
754. Specifically the whites dots are interpolated from the neighboring image pixels 
(directly from the photo sensors). 

Pixel Decimation 

Pixel decimation is a process that samples the signals from the photo 
sensors for low resolution needs. For example, not all the pixel signals from the image 
sensor are needed for a preview image on a small screen. Specifically, a small preview 
display screen located on digital camera system can only display a limited amount of data. 
Thus, it would be desirable to incorporate systems for reducing the amount of data that is 
output from the image sensor device in certain situations. 

Photo Sensor Skipping Pixel Decimation 

One method of reducing the amount of data that is output is to skip certain 
photo sensors during read out. According to one embodiment of the present invention, 
both the column and row decoders are configured to cause signals to be read out from 
every other photo sensors. Other repeated patterns (such as every third or every fourth 
photo sensor) can also be read out from the photo sensor array. 

Figure 8A illustrates an embodiment wherein the signals from an array of 
photo sensors 800 are being decimated by a factor of two. Through a set of predefined 
parameters, both colunrn and row decoders produce, respectively colunm and row address 
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signals 802 and 804. In other words, column and row address signals 802 and 804 
identify which signals from the array of photo sensors 800 are read out. As illustrated in 
Figure 8A, colunm and row address signals 802 and 804 cause signals from every other 
photo sensor to be read out to column buses 806. 

5 Figure 8B illustrates a conceptual diagram wherein only every other row 

and colunm from array 850 is activated during read out to produce a low-resolution pixel 
image 860. Numerically, the spatial resolution of the digital version 860 is one quarter of 
that of the array of photo sensors 850. Similarly, the readout speed is also improved four 
times. Those skilled understand that the decimation process is not possible in CCD 

P 10 image sensor in which all signals must be sequentially shifted out and subsequently stored 

iU in a large memory for decimation. 

«E One very important use for pixel decimation is digital camera parameter 

adjustment. As set forth in the previous section, window gating allows a small subarea to 
be used for auto focusing. Similarly, pixel decimation can be used to read out low 

i2 15 resolution images that sample that may be used to set the image capture parameters for a 

I , s 

digital camera. In one embodiment, the camera* s control system captures a decimated 
image that provides low-resolution pixel information from the entire pixel array area. 
The low-resolution image information is processed to determine the image quality. Based 
upon the -resolution image information, the camera's control system may adjust various 
20 camera parameters to improve the image quality. For example, the camera control system 
may adjust the white balance, the progranmiable gain amplifiers (PGA), the exposure 
time, the iris opening size, and/or other camera parameters in order to improve the quality 
of the final image to be captured. By decimating the image before processing, the camera 
control system has less data that must be processed for each parameter adjustment 
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iteration. By reducing the time per adjustment iteration, more iterations of parameter 
adjustment can be performed in the same amount of time. Thus, the camera control 
system can perform adjustments very quickly such that virtually no latency period exists 
between when the user points the camera and when the user can capture an image. 

Pixel Averaging Decimation 

Another method of performing pixel decimation is to combine neighboring 
pixels into a single pixel. For example, referring back to Figure 8B, the pixel processor 
can combine the four pixels in box 851 by averaging the pixel values to produce a single 
pixel output 861. The pixel averaging operation is similar to interpolation. However, 
unlike interpolation, the data from the original rows is then discarded. This example 
would be pixel averaging decimation by 1/4 since one pixel would be output for every 
four input pixels. Decimation can be performed up to 1/N since there are N rows in the 
pixel processor. Thus, if there were eight rows in the pixel processor, 1/8 pixel averaging 
decimation could be performed. 

Pixel Co-Siting 

Pixel Co-siting is a process that combines information from neighbor 
pixels to create a single pixel. Pixel Co-siting is a type of pixel interpolation that creates 
the same number of output pixels as there are input photo sensors. Pixel co-siting cures 
some image artifacts by smoothing the image. Figure 9A conceptually illustrates a small 
section of a Bayer pattern pixel array. The first row consists of alternating Red (R) and 
Green (G) pixels. The second row consists of alternating Green (G) and Blue (B) pixels. 
The remaining rows follow this pattem of alternating Red (R) and Green (G) pixel rows 
and alternating Green (G) and Blue (B) pixel rows. 
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If the first row were to be read out individually, it would not have any blue 
pixel data. Similarly, if the second row were to be read out individually, it would not 
have any blue pixel data. Instead, the pixels are "co-sited" wherein adjacent row and 
column data are combined form a single pixel. 

For example, referring to Figure 9A, the top two rows of the two leftmost 
colunms are combined to form a single pixel. Specifically, the photo sensors in box 904 
are combined to create pixel 905. Note that the pixel processor may average the value of 
the two green pixels within box 904. An adjacent pixel would be formed from the top 
two rows of the second and third columns. Specifically, the photo sensors in box 908 are 
combined to create pixel 909. In this manner, every pixel will be given Red, Green, and 
Blue pixel values. Although this example performs pixel co-siting with four photo 
sensors, pixel co-siting can be performed with other numbers of pixels. For example. 
Figure 9B illustrates a pixel co-siting example with nine neighboring pixels. 

Additional Features in Pixel Processor 

To reduce the design complexity and to increase the performance of an 
imaging system employing the current CMOS image sensor, additional features may be 
implemented in the pixel processor. 

In a digital camera, the conmion image formats of the output are preferably 
the Graphic Interchange Format (GIF) and the Graphic Interchange Format (GIF). In 
video, the common image formats of sequence of images include MPEG (Moving Picture 
Experts Group) and some emerging standards for video stream. All of these image 
formats are supported by the World Wide Web protocol, such as HTTP. Conventionally, 
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the format transformation from the pixel signals to one of the commonly used formats is 
carried out in a separate processor in the imaging system (e.g. the digital or video 
camera). For example, many imaging systems use a desktop personal computer along 
with an appropriate compression program. Such existing approaches present either 
considerable latency that may be experienced by a user or a complicated configuration. 

According to one embodiment, the image sensor 300 has 1024 by 768 
photo sensors that could produce 786,431 pixel charge signals. When the charge signals 
are digitized to the 10-bit precision, there would be nearly one megabyte of image data. 
Together with a sequence of images for video applications, the amount of data would be 
too big to be used and must be compressed before an apparatus coupled thereto can 
accommodate them. 

As described above, the on-chip memory 317 of Figure 3A may be loaded with 
instructions (microcode) that perform a pixel processing application. In one embodiment, 
a pixel processor program is executed in the pixel processor 318 to perform an image 
format transformation such that the output of the image sensor device 300 is in 
conformation with an image standard. The exact implementation of a compression 
program is not provided herein to avoid obscuring aspects of the present invention. 
Those skilled in the art appreciate that such microcode is available for general-purpose 
digital signal processors (DSP). The important feature of compressing pixel signals 
within an image sensor, contrary to those general-purpose DSPs, is that there are no 
additional memory needed to pre-store the pixel signals. The pixel signals from the ADC 
bank are compressed as they are coming. As a result, the overall signal throughput of the 
imaging system is not degraded due to the compression procedure in addition to the 
elimination of a large memory. 
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Furthermore, the pixel processor may be configured to output only black 
and white pixel signals. Referring to Figure 10, pixel signals 1002 from the R, G, and B 
photo sensors in the array of photo sensors are in a color format. Thus, signals from each 
group of the R, G, and B photo sensors compose a red, green, or blue color pixel, 
5 respectively. There several ways to intemally convert the signals from the R, G, and B 
photo sensors to gray-scale intensity signals. 

One of the methods is to apply different weights to the different 
components; for example, 0.299R, 0.587G and 0.114B. The sum of the weighted color 
components result in a gray-scale intensity value. 

10 Another way as illustrated in Figure 10, a group of four signals from a 

geometric group 1004 of the photo sensors are averaged to produce a gray-scale intensity 
value 1006. Therefore, the pixel signals are decimated by a factor of two while being 
converted to gray-scale intensity signals. Alternatively, each signal from a photo sensor 
1008 or 1010 is weighted differently and directly output as a gray-scale intensity 1012 

15 and 1014 to preserve the original spatial resolution. 

Another way is to simply output the values from the green photo sensors in 
Bayer pattern array. This method works well since the green sensor captures the most 
light and the human eye is most tuned to green light. 

The present invention has been described in sufficient detail with a certain 
20 degree of particularity. It is understood to those skilled in the art that the present 

disclosure of embodiments has been made by way of examples only. Numerous changes 
in the arrangement and combination of parts may be made without departing from the 
spirit and scope of the invention as claimed. Accordingly, the scope of the present 
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invention is defined by the appended claims rather than the forgoing description of 
embodiments. 
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